Using Theory of Change to design and evaluate complex mental health interventions in low and middle income countries: the case of PRIME

Background: Many health interventions are complex and consist of several interacting components (Medical Research Council 2008). These components include multiple causal strands, outcomes and levels of governance and may result in unexpected outcomes and non-linear change (Glouberman and Zimmerman 2002). As such they present challenges to the design and evaluation of complex health interventions. Although broad theoretical guidance has been developed by the UK Medical Research Council (MRC) (Craig, Dieppe et al. 2008), it contains little practical guidance and has been criticised for not including theory driven approaches to evaluation such as Theory of Change (ToC) (Anderson 2008). De Silva, Breuer et al. (2014) have proposed that ToC may complement the MRC guidance on complex health interventions. Methods: This study explores how ToC can strengthen the design and evaluation of complex health interventions using the example of The Programme for Improving Mental Health Care (PRIME). PRIME is a research programme which aimed to develop, implement and evaluate the integration of mental health into primary health in districts or sub-districts in Ethiopia, India, Nepal, South Africa and Uganda. A ToC approach was used in addition to other approaches to design and evaluate these complex mental health interventions. Firstly, I conducted a systematic review to determine the extent to which ToC has been used to design and evaluate public health interventions. Secondly, I compared the process of developing the ToC between all five PRIME countries and reflected on the value of ToC workshops using a framework analysis of workshop documentation and interviews with facilitators. Thirdly, I explored the development of the ToCs within the programme as a whole and the implications for the development of the intervention and the choice of evaluation methods. Fourthly, I presented a ToC for the integration of mental health care in low and middle income countries. Fifthly, I demonstrated how ToC can be used as a framework for a qualitative comparative analysis of process and outcome data using longitudinal data from 10 PRIME implementation facilities in Nepal. Lastly, I provide a set of 10 key lessons learned from PRIME in the application of ToC to complex mental health interventions. Results I found that the ToC approach has been used for the design and evaluation of public health interventions since the 1990s. However, there is a lack of clear description of the use of ToC in the literature on public health interventions and inconsistency in how it is used. In applying the ToC approach to PRIME, I found that facilitators reported that ToC workshops were a valuable way to develop ToCs and that different stakeholders at the workshop contribute different types of information to the ToC process. Hierarchies within the health system are an important consideration for ToC workshops as power dynamics are likely to influence the functioning of the group. In addition, I found that the development of a cross country ToC can result in a programme theory which is relevant for complex multilevel intervention in different contexts. This ToC can provide a framework to map contextually relevant interventions and can be used to complement other intervention development approaches. The ToC can also be used to ensure indicators for all the short-, medium- and long-term outcomes are identified. However, combining process and outcome data analysis using the ToC is not straightforward. Qualitative Comparative Analysis (QCA) can be used to analyse process and outcome data in a single analysis in health services research. Conclusion: ToC can be used to strengthen the design and evaluation of complex health interventions and can be used to complement the MRC guidance in the design and evaluation of complex health interventions

Theory of Change: a theory-driven approach to enhance the Medical Research Council's framework for complex interventions.

BACKGROUND: The Medical Research Councils' framework for complex interventions has been criticized for not including theory-driven approaches to evaluation. Although the framework does include broad guidance on the use of theory, it contains little practical guidance for implementers and there have been calls to develop a more comprehensive approach. A prospective, theory-driven process of intervention design and evaluation is required to develop complex healthcare interventions which are more likely to be effective, sustainable and scalable. METHODS: We propose a theory-driven approach to the design and evaluation of complex interventions by adapting and integrating a programmatic design and evaluation tool, Theory of Change (ToC), into the MRC framework for complex interventions. We provide a guide to what ToC is, how to construct one, and how to integrate its use into research projects seeking to design, implement and evaluate complex interventions using the MRC framework. We test this approach by using ToC within two randomized controlled trials and one non-randomized evaluation of complex interventions. RESULTS: Our application of ToC in three research projects has shown that ToC can strengthen key stages of the MRC framework. It can aid the development of interventions by providing a framework for enhanced stakeholder engagement and by explicitly designing an intervention that is embedded in the local context. For the feasibility and piloting stage, ToC enables the systematic identification of knowledge gaps to generate research questions that strengthen intervention design. ToC may improve the evaluation of interventions by providing a comprehensive set of indicators to evaluate all stages of the causal pathway through which an intervention achieves impact, combining evaluations of intervention effectiveness with detailed process evaluations into one theoretical framework. CONCLUSIONS: Incorporating a ToC approach into the MRC framework holds promise for improving the design and evaluation of complex interventions, thereby increasing the likelihood that the intervention will be ultimately effective, sustainable and scalable. We urge researchers developing and evaluating complex interventions to consider using this approach, to evaluate its usefulness and to build an evidence base to further refine the methodology. TRIAL REGISTRATION: Clinical trials.gov: NCT02160249

Using theory of change to design and evaluate public health interventions: a systematic review

Background: Despite the increasing popularity of the theory of change (ToC) approach, little is known about the extent to which ToC has been used in the design and evaluation of public health interventions. This review aims to determine how ToCs have been developed and used in the development and evaluation of public health interventions globally. Methods: We searched for papers reporting the use of “theory of change” in the development or evaluation of public health interventions in databases of peer-reviewed journal articles such as Scopus, Pubmed, PsychInfo, grey literature databases, Google and websites of development funders. We included papers of any date, language or study design. Both abstracts and full text papers were double screened. Data were extracted and narratively and quantitatively summarised. Results: A total of 62 papers were included in the review. Forty-nine (79 %) described the development of ToC, 18 (29 %) described the use of ToC in the development of the intervention and 49 (79 %) described the use of ToC in the evaluation of the intervention. Although a large number of papers were included in the review, their descriptions of the ToC development and use in intervention design and evaluation lacked detail. Conclusions: The use of the ToC approach is widespread in the public health literature. Clear reporting of the ToC process and outputs is important to strengthen the body of literature on practical application of ToC in order to develop our understanding of the benefits and advantages of using ToC. We also propose a checklist for reporting on the use of ToC to ensure transparent reporting and recommend that our checklist is used and refined by authors reporting the ToC approach

Evaluation of mhGAP training for primary healthcare workers in Mulanje, Malawi: a quasi-experimental and time series study

Abstract Background There has been a growing global movement championed by the World Health Organization (WHO) to integrate mental health into primary health care as the most effective way of reducing the mental health treatment gap. This study aimed to investigate the impact of WHO Mental Health Gap Action Programme (mhGAP) training and supervision on primary health workers’ knowledge, confidence, attitudes and detection rate of major mental disorders in Mulanje, Malawi. Method The study used a quasi-experimental method (single cohort pre- and post-measures) with an interrupted time-series design. A 2 day mhGAP training was delivered to 43 primary healthcare workers (PHWs) working in 18 primary care clinics serving the entire population of Mulanje, Malawi (population 684,107). Modules covered were psychosis, moderate-severe depression, and alcohol & substance use disorders. The PHWs completed pre and post-tests to assess knowledge, confidence and attitudes. Number of diagnosed cases was obtained from clinic registers for 5 months prior to and 7 months following training. Data was analyzed using mean scores, t-test, one-way analysis of variance and linear regression. Results The mean knowledge score increased significantly from 11.8 (SD: 0.33) before training to 15.1 (SD: 0.38) immediately after training; t (42) = 7.79, p < 0.01. Similarly, mean knowledge score was significantly higher 6 months post training at 13.9 (SD: 2.52) compared to before; t (42) = 4.57, p < 0.01. The mean confidence score also increased significantly from 39.9 (SD: 7.68) before training to 49.6 (SD: 06.14) immediately after training; t (84) = 8.43, p < 0.01. It was also significantly higher 6 months post training 46.8, (SD: 6.03) compared to before; t (84) = 6.60, p < 0.01. One-way analysis of variance showed no significant difference in mean scores on all four components of the scale used to measure attitudes. A significant positive change in the trend in mental health service utilization after the intervention was demonstrated using a segmented linear regression (β = 2.43 (95% CI 1.02; 3.83) as compared to before (β = − 0.22 (95% CI − 2.67; 2.23) and immediately after (β = 1.63 (95% CI − 7.31; 10.57). Conclusion The findings of this study add to the growing evidence for policy makers of the effectiveness of mhGAP training and supervision in a resource-constrained country

Developing a mental health care plan in a low resource setting: the theory of change approach

Abstract Background Scaling up mental healthcare through integration into primary care remains the main strategy to address the extensive unmet mental health need in low-income countries. For integrated care to achieve its goal, a clear understanding of the organisational processes that can promote and hinder the integration and delivery of mental health care is essential. Theory of Change (ToC), a method employed in the planning, implementation and evaluation of complex community initiatives, is an innovative approach that has the potential to assist in the development of a comprehensive mental health care plan (MHCP), which can inform the delivery of integrated care. We used the ToC approach to develop a MHCP in a rural district in Ethiopia. The work was part of a cross-country study, the Programme for Improving Mental Health Care (PRIME) which focuses on developing evidence on the integration of mental health in to primary care. Methods An iterative ToC development process was undertaken involving multiple workshops with stakeholders from diverse backgrounds that included representatives from the community, faith and traditional healers, community associations, non-governmental organisations, Zonal, Regional and Federal level government offices, higher education institutions, social work and mental health specialists (psychiatrists and psychiatric nurses). The objective of this study is to report the process of implementing the ToC approach in developing mental health care plan. Results A total of 46 persons participated in four ToC workshops. Four critical path dimensions were identified: community, health facility, administrative and higher level care organisation. The ToC participants were actively engaged in the process and the ToC encouraged strong commitment among participants. Key opportunities and barriers to implementation and how to overcome these were suggested. During the workshops, a map incorporating the key agreed outcomes and outcome indicators was developed and finalized later. Conclusions The ToC approach was found to be an important component in the development of the MHCP and to encourage broad political support for the integration of mental health services into primary care. The method may have broader applicability in planning complex health interventions in low resource settings

Active inclusion of people living with dementia in planning for dementia care and services in low- and middle-income countries

Involving people living with dementia in service design and planning has become more common in high-income countries. It remains rare in low- and middle-income countries where two-thirds of the world’s people with dementia live. In this commentary article, we explore the barriers to inclusion of people living with dementia in planning in low- and middle-income countries and make a case for the inclusion of people living with dementia in care and service planning. We suggest how this can be done at individual, community or national and state level using the following principles: 1) respecting the rights of people living with dementia to self-determination; 2) valuing people living with dementia’s unique understanding of dementia; 3) creating a culture of active inclusion which creates a space for people living with dementia to participate and 4) ensuring appropriate accommodations are in place to maximise participation

Using workshops to develop theories of change in five low and middle income countries: lessons from the programme for improving mental health care (PRIME)

Abstract Background The Theory of Change (ToC) approach has been used to develop and evaluate complex health initiatives in a participatory way in high income countries. Little is known about its use to develop mental health care plans in low and middle income countries where mental health services remain inadequate. Aims ToC workshops were held as part of formative phase of the Programme for Improving Mental Health Care (PRIME) in order 1) to develop a structured logical and evidence-based ToC map as a basis for a mental health care plan in each district; (2) to contextualise the plans; and (3) to obtain stakeholder buy-in in Ethiopia, India, Nepal, South Africa and Uganda. This study describes the structure and facilitator’s experiences of ToC workshops. Methods The facilitators of the ToC workshops were interviewed and the interviews were recorded, transcribed and analysed together with process documentation from the workshops using a framework analysis approach. Results Thirteen workshops were held in the five PRIME countries at different levels of the health system. The ToC workshops achieved their stated goals with the contributions of different stakeholders. District health planners, mental health specialists, and researchers contributed the most to the development of the ToC while service providers provided detailed contextual information. Buy-in was achieved from all stakeholders but valued more from those in control of resources. Conclusions ToC workshops are a useful approach for developing ToCs as a basis for mental health care plans because they facilitate logical, evidence based and contextualised plans, while promoting stakeholder buy in. Because of the existing hierarchies within some health systems, strategies such as limiting the types of participants and stratifying the workshops can be used to ensure productive workshops

Challenges and opportunities for implementing integrated mental health care: a district level situation analysis from five low- and middle-income countries.

BACKGROUND: Little is known about how to tailor implementation of mental health services in low- and middle-income countries (LMICs) to the diverse settings encountered within and between countries. In this paper we compare the baseline context, challenges and opportunities in districts in five LMICs (Ethiopia, India, Nepal, South Africa and Uganda) participating in the PRogramme for Improving Mental health carE (PRIME). The purpose was to inform development and implementation of a comprehensive district plan to integrate mental health into primary care. METHODS: A situation analysis tool was developed for the study, drawing on existing tools and expert consensus. Cross-sectional information obtained was largely in the public domain in all five districts. RESULTS: The PRIME study districts face substantial contextual and health system challenges many of which are common across sites. Reliable information on existing treatment coverage for mental disorders was unavailable. Particularly in the low-income countries, many health service organisational requirements for mental health care were absent, including specialist mental health professionals to support the service and reliable supplies of medication. Across all sites, community mental health literacy was low and there were no models of multi-sectoral working or collaborations with traditional or religious healers. Nonetheless health system opportunities were apparent. In each district there was potential to apply existing models of care for tuberculosis and HIV or non-communicable disorders, which have established mechanisms for detection of drop-out from care, outreach and adherence support. The extensive networks of community-based health workers and volunteers in most districts provide further opportunities to expand mental health care. CONCLUSIONS: The low level of baseline health system preparedness across sites underlines that interventions at the levels of health care organisation, health facility and community will all be essential for sustainable delivery of quality mental health care integrated into primary care

Psychosis Recovery Orientation in Malawi by Improving Services and Engagement (PROMISE) protocol

Malawi has a population of around 20 million people and is one of the world's most economically deprived nations. Severe mental illness (largely comprising psychoses and severe mood disorders) is managed by a very small number of staff in four tertiary facilities, aided by clinical officers and nurses in general hospitals and clinics. Given these constraints, psychosis is largely undetected and untreated, with a median duration of untreated psychosis (DUP) of around six years. Our aim is to work with people with lived experience (PWLE), caregivers, local communities and health leaders to develop acceptable and sustainable psychosis detection and management systems to increase psychosis awareness, reduce DUP, and to improve the health and lives of people with psychosis in Malawi. We will use the UK Medical Research Council guidance for developing and evaluating complex interventions, including qualitative work to explore diverse perspectives around psychosis detection, management, and outcomes, augmented by co-design with PWLE, and underpinned by a Theory of Change. Planned deliverables include a readily usable management blueprint encompassing education and community supports, with an integrated care pathway that includes Primary Health Centre clinics and District Mental Health Teams. PWLE and caregivers will be closely involved throughout to ensure that the interventions are shaped by the communities concerned. The effect of the interventions will be assessed with a quasi-experimental sequential implementation in three regions, in terms of DUP reduction, symptom remission, functional recovery and PWLE / caregiver impact, with quality of life as the primary outcome. As the study team is focused on long-term impact, we recognise the importance of having embedded, robust evaluation of the programme as a whole. We will therefore evaluate implementation processes and outcomes, and cost-effectiveness, to demonstrate the value of this approach to the Ministry of Health, and to encourage longer-term adoption across Malawi.</p